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DOCUMENT-IDENTIFIER: US 20020102022 Al 

TITLE: Detecting and utilizing add-on information from a scanned document image 



Summary of- Invention Paragraph : 

[0005] Moreover, there are also times when the person editing or annotating a 
printed document may make handwritten changes or additions to the document and then 
later need to have a copy of the document in its original, unedited and annotated 
form. Unless an extra copy of the document was previously saved, in its original 
form. Anyone who has ever tried to reconstitute a heavily edited document may 
manually erasing or covering up the edits and annotations understands how tedious 
and time consuming the process is. Automated methods of separating handwritten 
annotations from printed text, if developed, could potentially relieve much of the 
tedium. 
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Detail Description Paragraph : 

[0054] Starting at step 402, the secondary version of the document having the 
printed text lines and the handwritten annotations is scanned. The handwritten 
annotations are separated from the printed text lines according to the 
aforementioned method for detecting and' separating the add-on information 10 (FIG. 
1), step 404. The separated printed text lines are then compared to all the 
original documents in the database at step 406 in order to find the identical 
document, step 408. If there does not exist an original document for the printed 
text lines, the extracted printed text line is stored as original document, step 
410, and handwritten annotations is -stored in a different memory location, step 
412. The memory location of the stored handwritten annotation is linked to the 
memory location of the original document, step 414. The process is terminated. In 
case of extracted printed text lines having an original document in the database, 
only handwritten annotations are stored, step 416, and therefore linked to the 
identified original document, step 418. The process is terminated. In this case, 
the extracted printed text lines are ignored, and memory space is saved . 
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DOCUMENT-IDENTIFIER: US 20020079371 Al 
TITLE: Multi-moded scanning pen with feedback 



Detail Description Paragraph : 

[0163] If desired, the message and its pointers may be stored as part of the 
database 310 (FIG. 3); however, if full recognition has not been performed, it is 
likely that the system will not be able to index the information in any meaningful 
way without user intervention. Either the message as a whole, with pointers to 
interesting data, can be stored in the database, or only the name and number (for 
example, after the user has verified their correct extraction) can be selected for 
merger into the database. Accordingly, once extraction has taken place, the 
extracted number can be dealt with in at least three different ways: it may be 
saved as a full audio stream (much as pen computers save unrecognized handwriting ) 
and remain a transient form annotating the particular message; it may be saved to 
the database (with all or part of the greeting to identify the caller); or it can 
be recognized as numbers, and merged into the appropriate organizing construct 
(such as a calendar or electronic address book) . This technique can also be used as 
an accelerator—a key on the phone keypad may be used to skip directly to the 
embedded phone number in a long message. In this scheme, not only does the audio 
stream remain unchanged; it also remains in the same medium for access. 
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DOCUMENT- IDENTIFIER: US 20020056577 Al 
TITLE: Collaborative input system 



Detail Description Paragraph : 

[0024] A single, predefined coordinate system is used in the system 10 that resides 
on the host computer 12. All digitizers 18 are mapped in the same graphic display 
space that coincides with the display 14. For example, as shown in FIG. 1, 
information 25 input on digitizer 18' coincides with the display 14 so as to be 
shown in the corresponding location on the display 14 and on the projected image on 
the whiteboard 21. Handwritten information 25 does not appear on the digitizer 18, 
the stoke is however captured electronically and is represented graphically on the 
display 14 and whiteboard 21 (if provided) together with the image 17. The screen 
capture and . annotation is done by a software application such as the Interwrite . TM. 
software of GTCO CalComp. The software runs on the host computer 12 which has the 
drivers included for hardware to map the digitizer display space to the display 14. 
The software has two basic elements: 1) the application that allows the images to 
be captured, annotated and saved and 2) the interface to the hardware, that 
interprets the information coming in from the digitizers 18. 
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File: PGPB 
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DOCUMENT-IDENTIFIER: US 20020051262 Al 

TITLE: Image capture device with handwritten annotation 



Detail Description Paragraph : 

[0044] Continuing with the example, the user is preferably prompted to enter 
instructions in location 303 set aside for such entries. Four instructions 403 are 
shown having been "entered by the user, which are, from top to .bottom, " save file to 
accident-img, " "Attach to mail message," "mail to Dave, Larry, and Pete," and 
"place directed annotation at bottom center of image." Upon reviewing the user- 
entered instruction information the inventive scanner preferably performs 
handwriting analysis on the handwritten entries to convert the individual 
characters into to machine-generated characters. Thereafter, the inventive scanner 
preferably interprets the sequences of characters to correlate the user-entered 
sequence of characters with distinct commands recognizable to the scanner. The 
scanner then preferably executes the instructions in the order entered, unless an 
alternate order is indicated by the user. 
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TITLE: Distributed document -based calendaring system 



Detailed Description Text (61) : 

If desired, the message and its pointers may be stored as part of the database 310 
(FIG. 3); however, if full recognition has not been performed, it is likely that 
the system will not be able to index the information in any meaningful way without 
user intervention. Either the message as a whole, with pointers to interesting 
data, can be stored in the database, or only the name and number (for example, 
after the user has verified their correct extraction) can be selected for merger 
into the database. Accordingly, once extraction has taken place, the extracted 
number can be dealt with in at least three different ways: it may be saved as a 
full audio stream (much as pen computers save unrecognized handwriting ) and remain 
a transient form annotating the particular message; it may be saved to the database 
(with all or part of the greeting to identify the caller); or it can be recognized 
as numbers, and merged into the appropriate organizing construct (such as a 
calendar or electronic address book) . This technique can also be used as an 
accelerator — a key on the phone keypad may be used to skip directly to the embedded 
phone number in a long message. In this scheme, not only does the audio stream 
remain unchanged; it also remains in the same medium for access. 
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□ 7. Document ID: US 6651218 Bl 

L2: Entry 7 of 20 File: USPT Nov 18, 2003 

DOCUMENT-IDENTIFIER: US 6651218 Bl 

TITLE: Dynamic content database for multiple document genres 



Detailed Description Text (63):. 

If desired, the message and its pointers may be stored as part of the database 310 
(FIG. 3); however, if full recognition has not been performed, it is likely that 
the system will not be able to index the information in any meaningful way without 
user intervention. Either the message as a whole, with pointers to interesting 
data, can be stored in the database, or only the name and number (for example, 
after the user has verified their correct extraction) can be selected for merger 
into the database. Accordingly, once extraction has taken place, the extracted 
number can be dealt with in at least three different ways: it may be saved as a 
full audio stream (much as pen computers save unrecognized handwriting ) and remain 
a transient form annotating the particular message; it may be saved to the database 
(with all or part of the greeting to identify the caller) ; or it can be recognized 
as numbers, and merged into the appropriate organizing construct (such as a 
calendar or electronic address book) . This technique can also be used as an 
accelerator — a key on the phone keypad may be used to skip directly to the embedded 
phone number in a long message. In this scheme, not only does the audio stream 
remain unchanged; it also remains in the same medium for access. 
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L2: Entry .8 of 20 



File: USPT 



Jan 29, 2002 



DOCUMENT-IDENTIFIER: US 6342901 Bl 

TITLE: Interactive device for displaying information from multiple sources 



Detailed Description Text (98): 

If desired, the message and its pointers may be stored as part of the database 310 
(FIG. 3); however, if full recognition has not been performed, it is likely that 
the system will not be able to index the information in any meaningful way without 
user intervention. Either the message as a whole, with pointers to interesting 
data, can be stored in the database, or only the name and number (for example, 
after the user has verified their correct extraction) can be selected for merger 
into the database. Accordingly, once extraction has taken place, the extracted 
number can be dealt with in at least three different ways: it may be saved as a 
full audio stream (much as pen computers save unrecognized handwriting ) and remain 
a transient form annotating the particular message; it may be saved to the database 
(with all or part of the greeting to identify the caller) ; or it can be recognized 
as numbers, and merged into the appropriate organizing construct (such as a 
calendar or electronic address book) . This technigue can also be used as an 
accelerator--a key on the phone keypad may be used to skip directly to the embedded 
phone number in a long message. In this scheme, not only does the audio stream 
remain unchanged; it also remains in the same medium for access. 



DOCUMENT-IDENTIFIER: US 6249765 Bl 

TITLE: System and method for extracting data from audio messages 
Detailed Description Text (99): 

If desired, the message and its pointers may be stored as part of the database 310 
(FIG. 3); however, if full recognition has not been performed, it is likely that 
the system will not be able to index the information in any meaningful way without 
user intervention. Either the message as a whole, with pointers to interesting 
data, can be stored in the database, or only the name and number (for example, 
after the user has verified their correct extraction) can be selected for merger 
into the database. Accordingly, once extraction has taken place, the extracted 
number can be dealt with in at least three different ways: it may be saved as a 
full audio stream (much as pen computers save unrecognized handwriting ) and remain 
a transient form annotating the particular message; it may be saved to the database 
(with all or part of the greeting to identify the caller) ; or it can be recognized 
as numbers, and merged into the appropriate organizing construct (such as a 
calendar or electronic address book) . This technigue can also be used as an 
accelerator--a key on the phone keypad may be used to skip directly to the embedded 
phone number in a long message. In this scheme, not only does the audio stream 
remain unchanged; it also remains in the same medium for access. 
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□ 10. Document ID: US 5809498 A 

L2: Entry 10 of 20 



File: USPT 



Sep 15, 1998 



DOCUMENT-IDENTIFIER: US 5809498 A 

TITLE: Method of locating a penstroke sequence in a computer 



Detailed Description Text (15) : 

Images 34-37 include, with the pictographic names of the files, an icon of a 
letter, at 32, indicating to the user that the contents of the so -annotated files 
are letters. Pictographic names 34 and 35 have not only the same icon of a letter, 
at 32, but also have highly similar images, Josh and* Josh2 respectively. The user 
may opt to input and save a new handwritten representation of the contents of the 
file each time a related file is created; or, may use part of the originally 
created image when naming a subsequently created related file. In the latter 
manner, the original pictographic image is copied and altered, by erasure or by the 
addition of information, for example. Later retrieval of the related documents can 
be greatly enhanced irrespective of the specific matching algorithm which is 
employed, as will be further detailed below. 



Class if ication 



13 11. Document ID: US 5761485 A 

L2: Entry 11 of 20 File: USPT Jun 2, 1998 

DOCUMENT-IDENTIFIER: US 5761485 A 
TITLE: Personal electronic book system 

Detailed Description Text (30) : 

Embodiments of the Personal Electronic Book incorporating handwritten or 
handprinted note capability further comprise handwriting recognition software 
incorporated into the proprietary operating system. User input via one of the 
touchscreens is analyzed and converted into digital characters that are displayed 
for confirmation, a keyboard, or the like, can be displayed to enter corrections 
and then the annotation saved upon user command. Alternatively, Graphical software 
capabilities are added to the proprietary operating system that interprets the 
user's handwriting as a graphical image, stores this graphical image, and the 
displays this image along with the annotated document or in a user-created 
document . 
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DOCUMENT- IDENTIFIER: US 574 9908 A 

TITLE: Methods and apparatus for annotating data in an implantable device 
programmer using digitally recorded sound 

Detailed Description Text (95): 

Another alternative embodiment of the present invention is shown in FIG. 21. In 
this embodiment, the transcription of voice data is carried out using the digitizer 
pen 102 (FIG. 1) rather than using a keyboard. The elements displayed on the 
display screen in this embodiment preferably are the same as those shown in FIGS. 
17 and 18, and are thus labeled with the same reference numbers (except for the ink 
tool icon 450, text window 452, and text 454 as explained below). The like-numbered 
elements operate in the same manner as those described above in reference to FIGS. 
17 and 18 and will not be discussed further here. After selecting the voice 
annotation 410 for transcription, the medical specialist taps an ink tool icon 450 
to allow the pen 102 (FIG. 1) to write in the text window 452. When the medical 
specialist is finished writing t he or she taps the ink tool icon 450 again. The 
medical specialist saves handwritten text 454 by tapping the save button 438. In 
this embodiment, the display screen shown in FIG. 21 is displayed on the display 
screen of the tablet computer 100 so that the user can write with the digitizer pen 



DOCUMENT-IDENTIFIER: . US 5734882 A 

TITLE: Pictographic bitmap naming of files in pen-based computer systems 



Detailed Description Text (15) : 

Images 34-37 include, with the pictographic names of the files, an icon of a 
letter, at 32, indicating to the user that the contents of the so -annotated files 
are letters. Pictographic names 34 and 35 have not only the same icon of a letter, 
at 32, but also have highly similar images, Josh and Josh2 respectively. The user 
may opt to input and save a new handwritten representation of the contents of the 
file each time a related file is created; or, may use part of the originally 
created image when naming a subsequently created related file. In the latter 
manner, the original pictographic image is copied and altered, by erasure or by the 
addition of information, for example. Later retrieval of the related documents can 
be greatly enhanced irrespective of the specific matching algorithm which is 
employed, as will be further detailed below. 



102 (FIG. 1). 
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DOCUMENT-IDENTIFIER: US 5561446 A 

TITLE: Method and apparatus for wireless remote Information retrieval and pen-based 
data entry 

Detailed Description Text (16) : 

In addition to the handwritten notes corresponding to spaces on a standardized 
form, handwritten drawings may be entered on a template such as the one shown in 
FIG. 5. FIG. 5 shows how the interface 30 can display a diagram 60 of human 
skeletal muscles, for example, to aid in patient diagnosis. When such a template is 
displayed on the position sensitive display 12a, the user can enter drawings 62a 
and/or notes 62b using the stylus 12b thereby annotating the diagram 60. The 
annotated diagram 60 becomes a part of the patient records by saving the drawings 
and notes 62a, 62b as electronic ink as described in more detail below with 
reference to FIG. 7. 
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□ 15. Document ID: US RE34476 E 

L2: Entry 15 of 20 



File: USPT 



Dec 14, 1993 



DOCUMENT-IDENTIFIER: US RE34476 E 

TITLE: Hybrid information management system for handwriting and text 



Brief Summary Text (29) : 

The system is also designed strictly for one-time annotation or creation of an 
handwritten note. There is no capability for manipulating elements of handwritten 
material or for subsequent modification. Once the handwritten material is " saved ", 
i.e., committed to file storage status, it cannot be modified, only erased totally. 
This prevents the system from being used with information in an appointment book or 
"to-do" list format where the capability for updating an information element is 
critical. This limitation upon the manipulation of handwritten information elements 
also effectively prevents the facile organization and combination of multiple 
handwritten information elements which may be contained as sub-elements of other 
handwritten information sources into a combined handwritten document . Such a 
capability is key to assembly of a report, outline or paper drawing on separately 
stored information sub-elements (ideas, facts and other) . Because this kind .of 
information assembly is a very prominent information processing activity, it 
represents a severe limitation of the system. 

Detailed Description Text (93) : 

Extensive labor saving may also be realized through use of the Secretary module as 
an independent device. Its employment eliminates the following time consuming 
manual steps: physical transmittal of handwritten source copy; printing of one or 
more draft copies for editing/correction; return transmittal of the hand edited 
draft material; and copying of source and final documents for certain purposes. As 
compared with internal mail systems, electronic transmittal of handwritten source 
and annotated material results in a dramatic increase in throughput—a significant 
quality benefit. 
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□ 16. Document ID: US 5063600 A 



L2: Entry 16 of 20 



File: USPT 



Nov 5, 1991 



DOCUMENT- IDENTIFIER : US 5063600 A 

TITLE: Hybrid information management system for handwriting and text 



Brief Summary Text (28): 

The system is also designed strictly for one-time annotation or creation of an 
handwritten note. There is no capability for manipulating elements of handwritten 
material or for subsequent modification. Once the handwritten material is " saved ", 
i.e., committed to file storage status, it cannot be modified, only erased totally. 
This prevents the system from being used with information in an appointment book or 
"to-do" list format where the capability for updating an information element is 
critical. This limitation upon the manipulation of handwritten information elements 
also effectively prevents the facile organization and combination of multiple 
handwritten information elements which may be contained as sub-elements of other 
handwritten information sources into a combined handwritten document. Such a 
capability is key to assembly of a report, outline or paper drawing on separately 
stored information sub-elements (ideas, facts and other) . Because this kind of 
information assembly is a very prominent information processing activity, it 
represents a severe limitation of the system. 

Detailed Description Text (93) : 

Extensive labor saving may also be realized through use of the Secretary module as 
an independent device. Its employment eliminates the following time consuming 
manual steps: physical transmittal of handwritten source copy; printing of one or 
more draft copies for editing/correction; return transmittal of the hand edited 
draft material; and copying of source and final documents for certain purposes. As 
compared with internal mail systems, electronic transmittal of handwritten source 
and annotated material results in a dramatic increase in throughput--a significant 
quality benefit. 



DOCUMENT-IDENTIFIER: NA8911452 

TITLE: Linking Handwriting Annotation With Text 
Disclosure Text (1) : 

- Disclosed is a system that provides link and filing features for existing 
application programs with no functions for handwriting . It does not require 
application programs to be modified. This system is a terminate and stay resident 
(TSR) program and deals with all handwriting operations independent of the 
application program, a link feature preserves the correspondence between a text and 
its handwritten annotation . A filing feature automatically consolidates the 
annotated data into a single text file. Handwritten annotation means characters, 
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symbols, and figures, providing additional information on or beside text (Fig. 1) . 
- In order to preserve the correspondence between a text and its annotation, 
information on this correspondence has to be kept in the text data, because text is 
the only available data type for application programs, such as text editors and 
word processors. This information is pointed to by the "Link Marker". A link marker 
is a special type of character, but it is neither an alphabetic symbol nor a Kanji 
character. In order to establish a correspondence, the user specifies a group of 
strokes as a unit of a link operation and the location of linking. Then, the system 
puts the character code of the link marker into the text by simulating keyboard 
operations. Whenever the user moves or copies phrases or sentences that include the 
link marker to other locations, the corresponding annotations will follow the link 
marker. - The position of the link marker becomes the basis for displaying 
handwritten annotations on the text (Fig. 2) . The system searches for the link 
marker within the screen, which is the presentation space of the application. The 
application program converts the internal text format, which is unknown to this 
system, into a presentation style, such as a screen. Therefore, it is easy to show 
and establish the correspondence in the presentation space. - A link marker has 
information on linked strokes, such as the link identification number, strokes, 
link type, marker type, and graphics attribute. A link identification number 
defines the character code of a link marker, which is uniquely assigned in each 
file. Strokes are a collection of strokes and a unit for linking. The link type 
means the way the annotation is linked to the text (point link - row and column 
position of point, region link - row and column position of left-top and right- 
bottom in a region, page link - page number, line link - page and line number) . 
Some link types, such as region link, require several link markers in order to 
specify the area. The marker type specifies a figure in the link marker and is 
shaped like a finger pointer to the center of gravity of the strokes. Graphics 
attributes, such as color, line type, line width, and zooming factor, are the 
attributes of strokes. - The filing feature automatically saves handwritten 
annotation data in a single file without a specific request from the user. All 
operations on a text file are monitored by the system, so that operations on 
annotation data can be synchronized with those on text data. Handwritten 
information is put at the end of the file, and the application program cannot 
detect its existence. 
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□ 18. Document ID: NN8307508 

L2: Entry 18 of 20 File: TDBD Jul 1, 1983 



DOCUMENT- IDENTIFIER: NN8307508 

TITLE: Enhanced Handwritten Annotation of Electronic Documents 
Disclosure Text (1) : 

- This article describes security enhancements to the handwritten annotation 
subsystem described in the preceding article. These enhancements enable a user of 
•the subsystem to specify the identity or the authorization level of the individual 
or individuals to whom the user's handwritten annotation is directed, and where 
appropriate, they also may qualify or restrict a user's ability to create or view 
annotations according to the identity or authorization level of the user. An 
authorized user also may query the subsystem for information regarding the origin 
of an annotation and the users who are permitted access to it. - The office system 
to which these enhancements relate is illustrated above and is basically the same 
system as that shown in the preceding article. To support these enhancements, the 
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system software must be capable of distinguishing user IDs (UIDs) and also the 
authorization levels (ALs) associated with users. Each user has a unique UID and a 
certain range or ranges of ALs, which together determine the document annotations 
to which the user can have access and the individuals or classes of individuals to 
whom he can direct annotations. - In the present enhancement version, the 
annotation subsystem, comprising the interface and the associated save task and 
tablet task, has a modified "create an annotation " function and two new query 
functions, the purpose of which is explained below. Each of these modified and new 
functions entails a control upon the passage of annotation information in 
accordance with the UID or AL of the user or the recipient (s) to whom his 
annotation is directed. In the document structure, as presently constituted; the 
information for each annotation includes the originator UID, the date and time of 
origination, and the recipient UID or AL. - The effects of some of the commands 
described in the preceding article have been modified in the present version, as 
indicated below: Initialize annotation subsystem: The application provides the UID' 
and the user's authorization range (s), as well as the mapping information and 
display plane. The interface sets up the tablet and save tasks, and saves the UID 
for use in annotation creation. It checks the UID and authorization range (s) 
against the recipient information of all existing annotations as their information 
is read from the data area. Only those annotations which the user is authorized to 
see are included in the list which the interface maintains. - Create an annotation : 
The user must in some manner indicate the specific individual or class of 
individuals (by UID or AL) to which the annotation is directed. The application 
supplies this recipient UID or AL to the interface and indicates the display area 
valid for annotation . The interface places the tablet task in the annotation mode, 
creates a directory entry, and allocates space for the annotation in the document. 
It gets the time and date from the system, and saves them and recipient information 
as part of the annotation information. - The new query functions comprise "query 
and annotation " and "query all annotations". The "query and annotation " function 
supplies to the user (if he is authorized to receive it) certain information 
regarding an existing annotation which is associated with either an identifying 
symbol in the displayed document or an identifying number in the range from "one" 
to the total number of annotations. Such information comprises the identity of the 
originator, the origination date and time, the AL or UID to which the annotation is 
directed, and the location of the annotation in relation to the text. A query 
function does not give a user access to the annotation itself. Such access, when 
permitted by the user's ID or AL, can be had by utilizing one of the view, blank 
and delete functions described in the preceding article. If the user desires 
information regarding all annotations which he is authorized to receive, he employs 
the "query all annotations" function. - The query commands and their corresponding 
effects are described below: Query an annotation : The user identifies the 
annotation in which he is interested by selecting its symbol in the displayed 
document or by specifying the number of that annotation . The application gives the 
selected location or number against the list of locations or numbers for which the 
user is authorized to receive annotation information, and if a match is found, the 
query items are presented to the user. - Query all annotations: The user's request 
for all annotation information is passed by the application to the interface, which 
counts the number of annotations in the list of annotations which the user is 
authorized to receive and returns this number to the application. The application 
presents to the user a list of the query items for all annotations that exist 
within the user's authorization range(s). - The commands for viewing, blanking and 
deleting annotations, which are described in the preceding article, also can be 
employed in the present system, but since these commands work from the list of 
annotations maintained by the interface, only those annotations to which the user 
is authorized to have access can be viewed, blanked or deleted by him. 
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□ 19. Document ID: NN8307504 

L2: Entry 19 of 20 File: TDBD Jul 1, 1983 

DOCUMENT- IDENTIFIER: NN8307504 

TITLE: Handwritten Annotation of Electronic Documents 



Disclosure Text (1) : 

- This software subsystem enables handwritten annotations to be superimposed upon 
or deleted from electronically displayed documents using an electronic tablet as an 
entry device, so that the user of an electronic office system can annotate 
electronic documents by hand as readily as this can be done by a user of paper 
documents in a non- electronic office system. - The concept can best be implemented 
in a system having a multi- plane frame-buffered display which enables the document 
text and the annotations to be written on separate bit planes. It can be 
implemented also in a single-plane display system but with less functional 
convenience. The system also requires an electronic tablet which can provide 
information as to stylus or pen location and whether the pen is on or off the 
tablet while it is being moved. The term " annotation " is broadly used and may 
include signatures as well as hand-drawn figures and written notes. - Generally, a 
user does not want to view simultaneously all of the annotations to a document, 
written in full. In most instances it will suffice to indicate the existence of an 
annotation by placing a special symbol upon the displayed document at the place 
where the annotation is to appear, giving the user the capability of retrieving 
that annotation for full display merely by pointing to the symbol (through the 
medium of the pen and tablet) whenever he wants to view the annotation and perform 
operations upon it. The present system is designed on the assumption that the full 
display of annotations is only occasionally needed and that a symbolic display is 
usually adequate. - Referring to the block diagram, the handwritten annotation 
subsystem runs under control of the document review application task, and it 
comprises three software components: an interface and two high- priority tasks 
( save and tablet) running in conjunction with the application task. The interface 
component accepts commands from the application, interprets these commands, 
performs certain functions in response to them, and communicates with the save and 
tablet tasks. The. tablet task operates in either of two modes as chosen by the 
interface. In its selection mode the task only services the tablet, receiving 
information from the tablet and converting it to a form usable by the subsystem 
and/or the application. In its annotation mode the tablet task still services the 
tablet in accordance with the selection funcion, but also buffers the tablet 
information and draws it on the display when the user is writing in the appropriate 
area chosen by the application. The tablet task maintains two buffers for tablet 
information, and when one buffer is full, the tablet task enables the save task, 
which will store that buffer in the document while the other buffer is being filled 
with new information. - The application task sees the handwritten annotation 
subsystem as one or more procedural calls, corresponding to the commands the 
application receives from the user or generates itself. It is assumed herein that 
the electronic document is stored in two separate parts, namely, the text portion 
and the handwritten annotation portion. This allows the application to work only 
with the text portion and the annotation subsystem to work only with the annotation 
portion of the document. The annotation portion has two parts: a directory, which 
has an entry for each annotation, and a data area, which disregards how and where 
the annotation is to be drawn upon the document display. - The following is a 
description of various commands that may be utilized in the operations of the 
application task and the annotation subsystem, showing the responses of the system 
to each of these commands: Initialize annotation subsystem: The application 
provides information to map the tablet coordinate space into the display coordinate 
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space, and the plane on which to draw the annotations. The interface sets up the 
two subsystem tasks, placing the tablet task in the selection mode, and reads the 
document annotation directory to creat a list of annotations and their locations as 
read from the data area. - Display annotation symbols: The application indicates 
the portion of the document currently visible, thereby designating a display area 
valid for annotation . The interface checks the location of all annotations, one by 
one, and if the location of any annotation is within the valid area, the interface 
places an annotation symbol on the display at the location of that annotation . 
:AB .IN 0 Create an annotation : The application indicates a display area valid for 
annotation . The interface places the tablet task in the annotation mode", creates a 
directory entry, and allocates space for the annotation in the document. The 
annotation is drawn on the display by the user as he writes on the tablet, and is 
saved as part of the document. :AB .IN 0 End an annotation : The application gives 
nothing to the interface. The interface returns the tablet task to the selection 
mode and stores all pertinent information about the document in the data area. - 
View an annotation : As the user chooses an annotation to view by selecting its 
symbol in the displayed document, the application gives the user-selected location 
to the interface. The interface checks this location against its list of valid 
annotation locations within the permissible display area/ and if a match is found, 
the interface reads the information for that annotation and draws it on the 
display. - View all annotations: The application indicates a valid display area for 
annotations. The interface checks the locations of all annotations, and each time 
it finds a location within the valid area, it reads the information for the 
annotation located there and draws it on the display. - Blank an annotation : This 
command calls for a selected annotation to be blanked from the displayed document 
but not deleted from the stored document. The user selects the symbol of the 
annotation to be blanked. The application gives this location to the interface. The 
interface checks the location against its list of annotation locations, and if a 
match is found, it reads the information for that annotation and erases it from the 
display. - Blank all annotations: The application gives a valid display area to the 
interface. The interface checks all annotation locations, and each time it finds 
one within that area, it reads the information for that annotation and erases it 
from the display. - Delete an annotation : This command calls for a selected 
annotation to be both erased from the display and deleted from the document in 
storage. The user selects the symbol of the annotation to be deleted. The 
application gives this location to the interface, which checks it against the list 
of valid annotation locations. If a match is found, the interface reads the 
information for that annotation, erases the annotation from the display, deletes 
the annotation information from the data area, and deletes the directory entry. It 
also removes the annotation from its list and erases the annotation symbol, if 
necessary. - Delete all annotations: This calls for all annotations within the 
display area to be erased from the display and deleted from the document. The 
application gives a valid display area to the interface. The interface checks all 
annotation locations in its list, and each time it finds a location within the 
valid area, it reads the information for that annotation and erases it from the 
display. It then deletes the directory entries for all such annotations and clears 
them from the data area. It also removes these annotations from its list and erases 
the appropriate annotation symbols. 
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TITLE: User Interface for Audio Communication System. December 1982. 



Disclosure Text (1) : 

7p. This article describes a communication system using a telephone set which 
enables a user to compose, delete, move, and edit audio messages, audio memos, 
dictated letters and documents, and to distribute or receive the same from any 
other person or telephone set. The described system includes a special user 
interface, means for storing audio data in digital form, means for storing various 
commands for operation on the audio data, and a keyboard for commanding different 
functions upon the audio data and for entering strings of characters or commands 
associated with voice usage. Background to Problem. - Non-interactive 
communications provide a permanent record and have traditional distribution methods 
(e.g., U. S. Mail, public libraries), but these require very expensive, time 
consuming transcription (typing) and editing. Multiple copies are often required, 
and distribution is very slow. - Interactive communications have speed advantages 
(usually) , and allow voice intonation which carries much information. But 
interactive communication has become increasingly difficult. Studies show business 
people are unavailable over half the time, due to being on the telephone, out of 
their offices, and in meetings. Business people make 20 or so telephone calls each 
day, completing less than half of them to the party they want to talk with. They 
almost never leave a content message with a secretary, but rather a message to 
"call me back", and, of course, when the call-back is made, the same statistics 
apply. Thus, this call may not be completed. Because of frequent travel, both local 
and long distance, business people often do not know where the person is that they 
want to communicate with. Thus, locating people in time and space has become an 
increasingly improbable task. General Solution. - What is needed is a communication 
system that can be used directly by a person (without involving the cost and 
potential errors of a human intermediary specialist); that can be used from 
anywhere; that is sufficiently powerful to be useful by including the necessary 
broad range of communication needs; that communicates directly with people, not 
things (e.g., computer data bases) ; that handles soft information (opinions, 
attitudes, predictions, emotions) as well as hard information (facts, numbers, 
propositions) ; that handles information with a short life-time as well as a long 
life-time; that automatically updates itself; that is very easy to learn and easy 
to use; and that teaches people about itself. - Described herein are the best 
features of both non-interactive and interactive communication systems for speech 
filing. The system, called EASYCOMM, which is applicable to speech filing systems 
generally, allows a person to compose (edit, review, insert, delete, annotate, 
structure) a message, memo, • document ; to distribute this to any person in the 
world; and to receive a message, memo, or document from any person in the world. To 
do all this, EASYCOMM uses a push button telephone (or its equivalent) as the 
terminal. The user gives commands to EASYCOMM via the push button telephone keys. 
Documents are stored and modified in digital form, but can be received or played 
out in analog form or digital form. - A user interface, i.e., the language and 
feedback that the user and EASYCOMM use, was required to allow people to learn 
easily and remember how to use 12. keys to do any of the 100-200 different possible 
functions required (defined below) . The user-interface issue "was twofold: what are 
the useful functions needed in a communication system and, once identified, how 
should they map into the 12 keys of a push button telephone to be easily learned 
and remembered. Document Creation or Editing. - This article now describes what a 
person can do with EASYCOMM. A person, using only a push button telephone as a 
terminal, can create a message, memo, letter, or document; review and replace all 
or part of it; and insert, delete, move, and edit segments of it. (Hereafter 
messages, letters, memos, documents, etc., will be collectively referred to as 
"documents".) A person can annotate audio, written, printed, or graphic documents 
received from other people. While the present article emphasizes spoken documents, 
we have used EASYCOMM to compose and distribute handwritten and typed documents as 
well. Thus, this article covers other media, also. Unlike a tape-recorder or 
conventional dictating system, EASYCOMM is voice-activated. A person does not turn 
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it off and on while recording. In effect, EASYCOMM records only when a person is 
talking. For subsequent ease of listening, as well, as storage compaction, pauses 
in between words are automatically deleted by EASYCOMM. Document Distribution. - A 
person can send a document to any person in the world that knows how to use a 
telephone. A person can send a document to one person, to several people at once, 
or to people on distribution lists (either made up by that person or provided by 
EASYCOMM). A person can page other people or radio other people. Documents can be 
sent by dialing either the name of the person or the person's telephone number. Of 
importance, a recipient can receive a message at any telephone number in the world. 
The sender need not know where a registered recipient is or will be. In the case of 
ambiguities with names (e.g., several Smiths) EASYCOMM asks the user which Smith. - 
When a message is sent, EASYCOMM calls the recipient and- notifies him or her about 
this. This call may be made at a time designated by EASYCOMM, or it may be made at 
times designated either by the sender or recipient. EASYCOMM will automatically re- 
try, should a t recipient be unavailable. A person can make his or her own phone ring 
at anytime. Thus, one may send himself wake-up calls or personal reminders. - A 
person may send a request to a distribution list of people, and as they respond, 
EASYCOMM recognizes ■ this and deletes their names from the list. EASYCOMM will make 
additional reminder calls to the remaining people, reminding them of overdue 
replies. - The concept of distribution lists is further enhanced by EASYCOMM, which 
stores organizational charts and automatically updates standard distribution lists. 
- A person can ask EASYCOMM if a particular individual has listened to a message 
that he or she sent that individual. While a person can verify through the U. S. 
Post Office's Registered Mail system whether someone at the recipient's (presumed) 
location has accepted a document, this, unlike EASYCOMM, does not tell whether the 
intended recipient has actually read (listened to) that document. Secondly, 
EASYCOMM provides this information immediately, whereas U. S. Registered Mail, for 
example, requires several days. - A person can send a document to a recipient and 
ask EASYCOMM to automatically notify him or her once the recipient has listened to 
it. - A person can cancel a document that he has already sent. If the document is 
sent to several people, and if one or more people have listened to it, EASYCOMM can 
be told whether to cancel the document for those other people. - A person can amend 
an already sent document. Again, EASYCOMM has the ability to either allow this or 
not depending upon whether the original document was already heard by one or more 
people. - A person can, solicit a reply to his document, and both the sender and 
recipient can be automatically notified the day this reply is due, as well as if it 
is overdue. Document Filing and Retrieval. - A permanent record is made of each 
audio document, together with relevant useful statistics (e.g., time and date of 
creation, author's name, list of recipients, when each has heard it, actions they 
have taken on it) . These statistics automatically provide a chronological file that 
a person can easily search. The document itself can be erased by the recipient. 
EASYCOMM also erases documents that have been saved past a certain time. This time 
is set by a "customer" (i.e., the owner of the entire system). Retention dates can 
be specified by the sender also. EASYCOMM provides an audit trail that may be 
retained indefinitely. - A recipient can classify or file a document under any 
heading he or she chooses. The recipient can give the document a coded name as well 
as a spoken name. - A person can retrieve a document by sender's name,. by 
recipient's name, by file name, by date of origin, by classification (e.g., 
personal confidential, secret). Message Control. - A person has great control over 
the document he or she sends. A person can classify a document (e.g. , 
confidential, secret, personal) . A person can regulate what others may do with the 
document (e.g., listen only, listen and comment on it, listen, comment, and forward 
it, etc.). A person can control the retention period of his document. A person can 
cancel or amend an already sent document. A person can request responses to his 
document. Personalization. - A person can tailor EASYCOMM to his or her own needs. 
A person can set the amount of feedback from EASYCOMM, perhaps based upon how 
experienced he or she is in using EASYCOMM. A person can assign functions to 
special passwords. For example, a person may want his own spouse or family to have 
access only to the documents specifically for them (e.g., "I'll be home late 
tonight."). A person may want to allow his or her secretary to have access to all 
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non-conf.idential documents. These are accomplished through a Family Password and 
Secretary Password, respectively. While EASYCOMM provides each user with standard 
defaults for these passwords, the user can change them. - There are several user 
interface languages, and the user can select which one he or she uses. A new user, 
for example, may choose to use a system-driven, prompted language that experience 
has shown requires no user training. The experienced user, on the other hand, may 
choose a generative language. In addition, EASYCOMM has a facility that 
automatically monitors user behavior, and provides the feedback and language 
appropriate to the user's knowledge. Furthermore, self help tools on EASYCOMM are 
simply called by the user. - All of these functions are accomplished by a person 
pressing keys on, and recurring feedback from, a standard push button telephone. 
Experience has shown that this person requires only a little bit of training to do 
work of this function. - EASYCOMM' s features are now described. EASYCOMM can be 
used from anywhere. Unlike present telephone recording systems, a person can 
compose a document from any telephone and send it to other people without knowing 
their whereabouts or telephone numbers. Unlike present interactive computer systems 
that use typewriter or CRT display terminals, a person need not seek out a special 
purpose terminal, but can use the ubiquitous telephone. - EASYCOMM presents to the 
user stored voice prompts, hints, suggestions, and menu choices for accomplishing 
useful work. - EASYCOMM presents to the user stored voice confirming messages 
following appropriate key presses, stored voice error messages, warnings, or 
helpful hints following inappropriate or potentially inappropriate key presses. 
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